A2 Dashboard Workflow

Workflow for building and deploying interactive visualizations

Let's say you want to make it easy to explore some dataset, i.e.:

  • Make a visualization of the data
  • Maybe add some custom widgets to see the effects of some variables
  • Then deploy the result as a web app.

You can definitely do that in Python, but you would expect to:

  • Spend days of effort to get some initial prototype working in a Jupyter notebook, every time
  • Work hard to tame the resulting opaque mishmash of domain-specific, widget, and plotting code
  • Start over nearly from scratch whenever you need to:
    • Deploy in a standalone server
    • Visualize different aspects of your data
    • Scale up to larger (>100K) datasets

Step-by-step data-science workflow

Here we'll show a simple, flexible, powerful, step-by-step workflow, explaining which open-source tools solve each of the problems involved:

  • Step 1: Get some data
  • Step 2: Prototype a plot in a notebook
  • Step 3: Model your domain
  • Step 4: Get a widget-based UI for free
  • Step 5: Link your domain model to your visualization
  • Step 6: Widgets now control your interactive plots
  • Step 7: Deploy your dashboard
In [1]:
import holoviews as hv, geoviews as gv, param, dask.dataframe as dd, cartopy.crs as crs

from colorcet import cm, fire
from holoviews.operation import decimate
from holoviews.operation.datashader import datashade
from holoviews.streams import RangeXY
from geoviews.tile_sources import EsriImagery

Step 1: Get some data

  • Here we'll use a subset of the often-studied NYC Taxi dataset
  • About 12 million points of GPS locations from taxis
  • Stored in the efficient Parquet format for easy access
  • Loaded into a Dask dataframe for multi-core
    (and if needed, out-of-core or distributed) computation
In [2]:
%time df = dd.read_parquet('../data/nyc_taxi_wide.parq').persist()
print(len(df))
df.head(2)
CPU times: user 2.99 s, sys: 105 ms, total: 3.1 s
Wall time: 3.21 s
50001
Out[2]:
tpep_pickup_datetime tpep_dropoff_datetime passenger_count trip_distance pickup_x pickup_y dropoff_x dropoff_y fare_amount tip_amount dropoff_hour pickup_hour
0 2015-01-15 19:05:39 2015-01-15 19:23:42 1 1.59 -8236963.0 4975552.5 -8234835.5 4975627.0 12.0 3.25 19 19
1 2015-01-10 20:33:38 2015-01-10 20:53:28 1 3.30 -8237826.0 4971752.5 -8237020.5 4976875.0 14.5 2.00 20 20

Step 2: Prototype a plot in a notebook

  • A text-based representation isn't very useful for big datasets like this, so we need to build a plot
  • But we don't want to start a software project, so we use HoloViews:
    • Simple, declarative way to annotate your data for visualization
    • Large library of Elements with associated visual representation
    • Elements combine (lay out or overlay) easily
  • And we'll want live interactivity, so we'll use a Bokeh plotting extension
  • Result:
In [3]:
hv.extension('bokeh')
In [4]:
points = hv.Points(df, ['pickup_x', 'pickup_y'])
decimate(points)
Out[4]:

Here Points declares an object wrapping df , visualized as a scatterplot of the pickup locations. decimate limits how many points will be sent to the browser so it won't crash.

As you can see, HoloViews makes it simple to pop up a visualization of your data, getting something on screen with only a few characters of typing. But it's not particularly pretty, so let's customize it a bit:

In [5]:
opts = dict(width=700, height=600, xaxis=None, yaxis=None, bgcolor='black')
decimate(points.options(**opts))
Out[5]:

That looks a bit better, but it's still decimating the data nearly beyond recognition, so let's try using Datashader to rasterize it into a fixed-size image to send to the browser:

In [6]:
taxi_trips = datashade(points, cmap=fire).options(**opts)
taxi_trips
Out[6]:

Ok, that looks good now; there's clearly lots to explore in this dataset. Notice that the aspect ratio changed because Datashader is using every point, including some distant outliers. One way to fix the aspect ratio is to indicate that it's geographic data by overlaying it on a map:

In [7]:
taxi_trips = datashade(points, x_sampling=1, y_sampling=1, cmap=fire).options(**opts)
EsriImagery * taxi_trips
Out[7]:

We could add lots more visual elements (laying out additional plots left and right, overlaying annotations, etc.), but let's say that this is our basic visualization we'll want to share.

To sum up what we've done so far, here are the complete 10 lines of code required to generate this geo-located interactive plot of millions of datapoints in Jupyter:

import holoviews as hv, geoviews as gv, dask.dataframe as dd
from colorcet import fire
from holoviews.operation.datashader import datashade
from geoviews.tile_sources import EsriImagery
hv.extension('bokeh')
df = dd.read_parquet('../data/nyc_taxi_wide.parq').persist()
opts = dict(width=700, height=600, xaxis=None, yaxis=None, bgcolor='black')
points = hv.Points(df, ['pickup_x', 'pickup_y'])
taxi_trips = datashade(points, x_sampling=1, y_sampling=1, cmap=fire).options(\*\*opts)
EsriImagery * taxi_trips

Step 3: Model your domain

Now that we've prototyped a nice plot, we could keep editing the code above to explore this data. But at this point we will instead often wish to start sharing our results with people not familiar with programming visualizations in this way.

So the next step: figure out what we want our intended user to be able to change, and declare those variables or parameters with:

  • type and range checking
  • documentation strings
  • default values

The Param library allows declaring Python attributes having these features (and more, such as dynamic values and inheritance), letting you set up a well-defined space for a user (or you!) to explore.

NYC Taxi Parameters

In [8]:
cmaps = ['bgy','bgyw','bmw','bmy','fire','gray','kbc','kgy']

class NYCTaxiExplorer(param.Parameterized):
    alpha       = param.Magnitude(default=0.75, doc="Alpha value for the map opacity")
    plot        = param.ObjectSelector(default="pickup", objects=["pickup","dropoff"])
    colormap    = param.ObjectSelector(default='fire', objects=cmaps)
    passengers  = param.Range(default=(0, 10), bounds=(0, 10), doc="""
        Filter for taxi trips by number of passengers""")

Each Parameter is a normal Python attribute, but with special checks and functions run automatically when getting or setting.

Parameters capture your goals and your knowledge about your domain, declaratively.

Class level parameters

In [9]:
NYCTaxiExplorer.alpha
Out[9]:
0.75
In [10]:
NYCTaxiExplorer.alpha = 0.5
NYCTaxiExplorer.alpha
Out[10]:
0.5

Validation

In [11]:
try:
   NYCTaxiExplorer.alpha = '0'
except Exception as e:
    print(e)
    
try:
   NYCTaxiExplorer.passengers = (0,100)
except Exception as e:
    print(e) 
Parameter 'alpha' only takes numeric values
Parameter 'passengers' upper bound must be in range [0, 10]

Instance parameters

In [12]:
explorer = NYCTaxiExplorer(alpha=0.6)
explorer.alpha
Out[12]:
0.6
In [13]:
NYCTaxiExplorer.alpha
Out[13]:
0.5

Step 4: Get a widget-based UI for free

  • Parameters are purely declarative and independent of any widget toolkit, but contain all the information needed to build interactive widgets
  • Panel can generate UIs in Jupyter or Bokeh Server from Parameters
  • Declaration of parameters is independent of the UI library used
In [14]:
import panel as pn

pn.Row(NYCTaxiExplorer)
Out[14]:
In [15]:
NYCTaxiExplorer.passengers
Out[15]:
(0, 10)

We've now defined the space that's available for exploration, and the next step is to link up the parameter space with the code that specifies the plot:

In [16]:
class NYCTaxiExplorer(param.Parameterized):
    alpha       = param.Magnitude(default=0.75, doc="Alpha value for the map opacity")
    colormap    = param.ObjectSelector(default='fire', objects=cmaps)
    plot        = param.ObjectSelector(default="pickup", objects=["pickup","dropoff"])
    passengers  = param.Range(default=(0, 10), bounds=(0, 10))

    def make_view(self, x_range=None, y_range=None, **kwargs):
        points = hv.Points(df, kdims=[self.plot+'_x', self.plot+'_y'], vdims=['passenger_count'])
        selected = points.select(passenger_count=self.passengers)
        taxi_trips = datashade(selected, x_sampling=1, y_sampling=1, cmap=cm[self.colormap],
            width=800, height=475)
        return EsriImagery.clone(crs=crs.GOOGLE_MERCATOR).options(alpha=self.alpha, **opts) * taxi_trips

Note that the NYCTaxiExplorer class is entirely declarative (no widgets), and can be used "by hand" to provide range-checked and type-checked plotting for values from the declared parameter space:

In [17]:
explorer = NYCTaxiExplorer(alpha=0.4, plot="dropoff")
explorer.make_view()
Out[17]:

Step 6: Widgets now control your interactive plots

But in practice, why not pop up the widgets to make it fully interactive?

In [18]:
explorer = NYCTaxiExplorer()
r = pn.Row(explorer, explorer.make_view)
r
Out[18]:

Step 7: Deploy your dashboard

Ok, now you've got something worth sharing, running inside Jupyter. But if you want to share your interactive app with people who don't use Python, you'll now want to run a server with this same code.

  • Deploy with Bokeh Server :
    • Write the above code to a file
    • Add r.server_doc(); to the end to tell Bokeh Server which object to show in the dashboard
    • bokeh serve nyc_taxi/main.py

Complete dashboard code

In [19]:
with open('apps/nyc_taxi/main.py', 'r') as f: print(f.read())
import holoviews as hv, geoviews as gv, param, dask.dataframe as dd, cartopy.crs as crs
import panel as pn

from colorcet import cm
from holoviews.operation.datashader import rasterize, shade
from holoviews.streams import RangeXY

hv.extension('bokeh', logo=False)

usecols = ['dropoff_x','dropoff_y','pickup_x','pickup_y','dropoff_hour','pickup_hour','passenger_count']
df = dd.read_parquet('../../data/nyc_taxi_wide.parq')[usecols].persist()
opts = dict(width=1000,height=600,xaxis=None,yaxis=None,bgcolor='black',show_grid=False)
cmaps = ['fire','bgy','bgyw','bmy','gray','kbc']


class NYCTaxiExplorer(param.Parameterized):
    alpha      = param.Magnitude(default=0.75, doc="Alpha value for the map opacity")
    cmap       = param.ObjectSelector(cm['fire'], objects={c:cm[c] for c in cmaps})
    hour       = param.Range(default=(0, 24), bounds=(0, 24))
    location   = param.ObjectSelector(default='dropoff', objects=['dropoff', 'pickup'])

    @param.depends('location', 'hour')
    def points(self):
        points = hv.Points(df, kdims=[self.location+'_x', self.location+'_y'], vdims=['dropoff_hour'])
        if self.hour != (0, 24): points = points.select(dropoff_hour=self.hour)
        return points

    @param.depends('alpha')
    def tiles(self):
        return gv.tile_sources.StamenTerrain.options(alpha=self.alpha, **opts)

    def view(self,**kwargs):
        points = hv.DynamicMap(self.points)
        agg = rasterize(points, x_sampling=1, y_sampling=1, width=600, height=400)
        stream = hv.streams.Params(self, ['cmap'])
        tiles = hv.DynamicMap(self.tiles)
        return tiles * shade(agg, streams=[stream])

taxi = NYCTaxiExplorer(name="NYC Taxi Trips")
pn.Row(taxi.param, taxi.view()).servable()

Branching out

The other sections in this tutorial will expand on steps in this workflow, providing more step-by-step instructions for each of the major tasks. These techniques can create much more ambitious apps with very little additional code or effort:


Right click to download this notebook from GitHub.